The present invention relates to video compression device in general and to video encoding and video/audio/data multiplexing devices in particular.
Methods for encoding an audio-visual signal are known in the art. According to these methods, a video signal is digitized, analyzed and encoded in a compressed manner. These methods are implemented in computer systems, either in software, hardware or combined software-hardware forms.
Most hardware encoding systems consist of a set of semiconductor circuits arranged on a large circuit board. State of the art encoding systems include a single semiconductor circuit. Such a circuit is typically based on a high-power processor.
Reference is now made to FIG. 1, which is a block diagram illustration of a prior art video encoding circuit 10.
Encoding circuit 10 includes a video input processor 12, a motion estimation processor 14, a digital signal processor 16 and a bitstream processor 18. Processors 12-18, respectively, are generally connected in series.
Video input processor 12 captures and processes a video signal, and transfers it to motion estimation processor 14. Motion estimation processor 14 analyzes the motion of the video signal, and transfers the video signal and its associated motion analysis to digital signal processor 16. According to the data contained within the associated motion analysis, digital signal processor 16 processes and compresses the video signal, and transfers the compressed data to bitstream processor 18. Bitstream processor 18 formats the compressed data and creates therefrom an encoded video bitstream, which is transferred out of encoding circuit 10.
It will be appreciated by those skilled In the art that such an encoding circuit has several disadvantages. For example, one disadvantage of encoding circuit 10 is that bitstream processor 18 transfers the encoded video bitstream, data word by data word, directly to an element external to encoding circuit 10. Accordingly, each time such data word is ready, the encoded video data word is individually transferred to the external element. Transfer of the encoded video in such a fashion greatly increases the data traffic volume and creates communication bottlenecks in communication lines such as computer buses. Additionally, circuit 10 requires a dedicated storage/bus which is allocated on a full time basis, hence, magnifying these disturbances.
Another disadvantage is that encoding circuit 10 is able to perform the encoding of video signals, only. Usually, moving picture compression applications include multiframe videos and their associated audio paths. While the encoding circuit 10 performs video compression and encoding, the multiplexing of compressed video, audio and user data streams are performed separately. Such an approach increases the data traffic in the compression system and requires increased storage and processing bandwidth requirements, thereby greatly increasing the overall compression system complexity and cost.
Reference is now made to FIG. 2, which is a block diagram of a prior art video input processor 30, as may be typically included in encoding circuit 10. Video input processor 30 includes a video capture unit 32, a video preprocessor 34 and a video storage 36. The elements are generally connected in series.
Video capture unit 32 captures an input video signal and transfers it to video preprocessor 34. Video preprocessor 34 processes the video signal, including noise reduction, image enhancement, etc., and transfers the processed signal to the video storage 36. Video storage 36 buffers the video signal and transfers it to a memory unit (not shown) external to video input processor 30.
It will be appreciated by those skilled in the art that such video input processor has several disadvantages. For example, one disadvantage of processor 30 is that it does not perform image resolution scaling. Accordingly, only original resolution pictures can be processed and encoded.
Another disadvantage is that processor 30 does not perform statistical analysis of the video signal, since in order to perform comprehensive statistical analysis a video feedback from the storage is necessary, thus allowing interframe (picture to picture) analysis, and processor 30 is operable in xe2x80x9cfeed forwardxe2x80x9d manner, only. Accordingly, video input processor 30 can not detect developments in the video contents, such as scene change, flash, sudden motion, fade in/fade out etc.
Reference is now made to FIG. 3 which is a block diagram illustration of a prior art video encoding circuit 50, similar to encoding circuit 10, however, connected to a plurality of external memory units. As an example, FIG. 3 depicts circuit 50 connected to a pre-encoding memory unit 60, a reference memory unit 62 and a post-encoding memory unit 64, respectively. Reference is made in parallel to FIG. 4, a chart depicting the flow of data within circuit 50.
Encoding circuit 50 includes a video input processor 52, a motion estimation processor 54, a digital signal processor 56 and a bitstream processor 58. Processors 54 to 58, respectively, are generally connected in series.
In the present example, video encoding circuit 50 operates under MPEG video/audio compression standards. Hence, for purposes of clarity, reference to a current frame refers to a frame to be encoded. Reference to a reference frame refers to a frame that has already been encoded and reconstructed, preferably by digital signal processor 56, and transferred to and stored in reference memory unit 62. Reference frames are compared to current frames during the motion estimation task, which is generally performed by motion estimation processor 54.
Video input processor 52 captures a video signal, which contains a current frame, or a plurality of current frames, and processes and transfers them to external pre-encoding memory unit 60. External pre-encoding memory unit 60 implements an input frame buffer (not shown) which accumulates and re-orders the frames according to the standard required for the MPEG compression scheme.
External pre-encoding memory unit 60 transfers the current frames to motion estimation processor 54. External reference memory unit 62 transfers the reference frames also to motion estimation processor 54. Motion estimation processor 54, reads and compares both sets of frames, analyzes the motion of the video signal, and transfers the motion analysis to digital signal processor 56.
Digital signal processor 56 receives the current frames from the external pre-encoding memory 60, and according to the motion analysis received from motion estimation processor 54, processes and compresses the video signal. Digital signal processor 56 then transfers the compressed data to the bitstream processor 58. Digital signal processor 56 further reconstructs the reference frame and stores it in reference memory 62. Bitstream processor 58 encodes the compressed data and transfers an encoded video bitstream to external post-encoding memory unit 64.
It will be appreciated by those skilled in the art that such an encoding circuit has several disadvantages. For example, one disadvantage of encoding circuit 50 is that a plurality of separate memory units are needed to support its operations, thereby greatly increasing the cost and the complexity of any encoding system based on device 50.
Another disadvantage is that encoding circuit 50 has a plurality of separate memory interfaces. This increases the data traffic volume and the number of external connections of encoding circuit 50, thereby greatly increasing the cost and the complexity of encoding circuit 50. Another disadvantage is that encoder circuit 50 does not implement video and audio multiplexing, which is typically required in compression schemes.
Reference is now made to FIG. 5, a block diagram illustration of a typical interlaced formatted video in a normal encoding latency mode. The top line depicts the video fields before encoding, while bottom line depicts compressed frames after encoding.
Video Is generally received in a progressive or interlaced form. Typical interlaced rates are 60 fields/sec for NTSC standard and 50 fields/sec for PAL standard.
In order to minimize encoding latency, encoding circuits should begin processing of an image immediately after receipt of the minimal amount of image data. Video is comprised of a plurality of fields, wherein each frame has a top and bottom field, referenced herein as top m and bot m. The video fields illustrated in FIG. 5 are referenced top 0 and bot 0, top 1 and bot 1, etc. such that each pair of associated top and bot refers to a single frame.
Encoding circuits begin the encoding process after capturing M pictures, where M is defined as M=I/P ratio. I is defined as an I picture, which is the Intra frame or the first frame (frame 0) of the series of frames to be encoded, and P is a P picture, which is the predictive frame (frame 1), and is referenced from frame 0. The I/P ratio refers to a distance between successive I/P frames in video sequence. Typically, prior art encoding circuits, such as encoding circuit 10 or encoding circuit 50 begin processing the image after receipt of 2 or more pictures. Note that In FIG. 5, the I picture appears after the progression of 3 pictures, and as such, M=3.
It will be appreciated by those skilled in the art that such an encoding latency is a lengthy time period, and hence, has several disadvantages. One such disadvantage is that a large amount of storage is required to accumulate frames. Another disadvantage is that large latency does not enable use of encoding circuit 50 in time-sensitive interactive applications such as video conferencing and the like.
It is an object of the present invention to provide a novel device for encoding and multiplexing an audio-visual signal.
The applicants have realized that prior art encoding devices do not provide optimal division of the encoding task effort and hence, have longer than desired through-put time. As such, the present invention provides a novel buffer architecture and latency reduction mechanism for buffering uncompressed/ compressed information. The combination of the novel architecture, implemented with the latency reduction mechanism, provides for a proficient division of the encoding task effort and hence, a quicker through-put time.
In accordance with the present invention there is therefore provided a single chip digital signal processing device for real time video/audio compression. The device includes a plurality of processors, including a video input processor, a motion estimation processor, a digital signal processor, and a bitstream processor, wherein transfer of the signals within the device is done in a macroblock-by-macroblock manner, thus enabling pipeline macroblock-by-macroblock processing.
The video input processor receives, analyzes, scales and processes a digital signal. The motion estimation processor receives the processed signal, produces a motion analysis therefrom, and transfers the motion analysis to the digital signal processor. The digital signal processor, according to the motion analysis, compresses the processed signal and produces a compressed processed signal. A bitstream processor receives and formats the compressed processed signal.
Preferably, the device further includes a memory controller connected to the plurality of processors, wherein the memory controller controls data communication among the digital signal processor, the motion estimation processor, the video input processor and an external storage unit.
Additionally, preferably the device includes a multiplexing processor which multiplexes a plurality of digital signals and produces a multiplexed stream and a global controller which controls and schedules the video input processor, the motion estimation processor, the digital signal processor, the bitstream processor, the multiplexing processor and the memory controller.
Preferably, the motion estimation processor, the digital signal processor, the bitstream processor and the multiplexing processor operate in parallel. As such, the motion estimation processor operates on macroblock a of frame I, the digital signal processor operates on macroblock b of frame I, the bitstream processor operates on macroblock c of frame I, the multiplexing processor operates on frame J, wherein axe2x89xa7bxe2x89xa7c, and Ixe2x89xa7J.
The video input processor Includes a capture unit, an input video storage, a video storage, a pre-encoding processor, a scaler, a video processor and a controller,
The capture unit acquires a multiple frame video signal. The video storage buffers the multiple frame video signal thereby allowing adjustment between an internal video rate and an external data communication rate. The pre-encoding processor receives the multiple frame video signal from the capture unit and produces statistical analysis of the multiple frame video signal. The scaler receives the multiple frame video signal from the pre-encoding processor and modifies picture resolution. The video processor processes the multiple video signal. The controller controls and schedules the capture unit, the pre-encoding processor, the scaler, the video processor and the video storage. Alternatively, the input storage buffers the video signal thereby adjusting between an external communication rate and internal video rate.
Preferably, the multiple frame video signal is acquired from either a video interface or a host interface. Furthermore, the video input processor operates on frame K such that Kxe2x89xa7Ixe2x89xa7J.
In accordance with the present invention there is therefore provided a video compression system including a host interface, a memory unit and a digital signal processing device. The digital signal processing device receives a multiplicity of signals from the host interface and the memory unit and produces, in a pipeline macroblock-by-macroblock manner, a multiplexed encoded data stream.
Preferably, the multiplicity of signals include either a video signal, an audio signal, or a user data stream.
Preferably, the system additionally includes a video interface which supplies a video signal to the digital signal processing device. Alternatively, the system includes a compressed data interface which receives the encoded signal from the digital signal processing device and an audio interface which transfers a digitized audio/user data signal to the digital signal processing device.
In accordance with the present invention there is therefore provided a multiplexing processor which includes a first video storage, a second video storage, an audio/data storage, a processor and an output storage.
The first video storage buffers a compressed video bitstream, and transfers the compressed video bitstream to the external memory unit, thereby adjusting between internal video rate and external communication rate. The second video storage reads from the memory unit the compressed video bitstream, and buffers the compressed video bitstream, thereby adjusting between the external communication rate and the multiplexor processing rate rate.
The audio/data storage buffers the digitized audio/data signal and transfers the digitized audio/data signal to the processor, thereby adjusting between the external audio rate and the multiplexor processing rate. The processor connected to the first and second video storage, the audio/data storage and the output storage, and which produces a multiplexed video/audio data stream. The output storage buffers the multiplexed video/audio/data stream, thereby adjusting between multiplexed video/audio/data stream rate and external communication rate.
Preferably, the first video storage is connected to an external memory unit, wherein the first storage unit buffers the compressed video bitstream in a real time variable rate and transfers the compressed video bitstream in a burst to the memory unit.
Preferably, the second video storage transfers the compressed video bitstream in a real time variable rate to the processor, and wherein the external memory unit transfers the compressed video bitstream in a burst to the second video storage.
Additionally preferably, the audio/data storage is connected to an external audio source, wherein the audio/data storage transfers the digitized audio/data signal to the processor in a real time variable rate and the external audio source transfers the digitzed audio/data signal in a burst to the audio/storage storage.
The external memory unit acts as a temporary encoded video buffer, thereby accumulating compressed video when the processor is unable to accept the compressed video. The multiplexing processor interfaces directly with a variety of communication devices, each the variety of communication devices having a different communication speed, such as a computer bus, and an asynchronous transmission line.
The digital signal processing device includes a plurality of processors, wherein the plurality of processors includes a digital signal processor, a bitstream processor, a motion estimation processor, and alternatively, a video input processor and a multiplexing processor.
Preferably, the digital signal processing device further includes a memory controller connected to the plurality of processors, wherein the memory controller controls data communication among the digital signal processor, the motion estimation processor, the video input processor and an external storage unit. Alternatively, the device further includes a global controller which controls and schedules the video input processor, the motion estimation processor, the digital signal processor, the bitstream processor, the multiplexing processor and the memory controller.
There is therefore provided in accordance with the present invention a method for encoding, including the steps of capturing a pipeline of a multiplicity of digitized video frames and encoding the multiple digitized video frames, one macroblock at a time.